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THE  PRIMARY  OBJECTIVE  OF  SPRUCE  IS  TO  ADDRESS  THE  TECHNOLOGY  TRANSITION  PROBLEM  AND  BRIDGE 
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Executive  Summary 

The  Systems  and  Software  Producibility  Collaboration  and 
Experimentation  Environment  (SPRUCE^)  is  an  open  web  portal 
to  bring  together  DoD  software  developers,  users,  and  software 
engineering  researchers  virtually  by  enabling  their  collaboration 
on  specifying  and  solving  software  producibility  challenge 
problems.  SPRUCE  is  based  on  the  premise  that  well  articulated 
and  bounded  problems  can  spark  scientific  and  engineering 
innovation  in  software  producibility  and  help  to  bridge  the 
gap  between  technology  users  and  technology  providers. 
Key  SPRUCE  features  are:  self-organizing  communities  of 
interest  (Col),  dynamically  evolving  challenge  problems  with 
accompanying  artifacts,  and  built-in  experimentation  facilities 
to  reproduce  the  problems  and  evaluate  solution  benchmarks. 
To  participate  in  SPRUCE,  visit  www.sprucecommunity.org  2in(i 
request  an  account. 

Why  SPRUCE? 

Consider  an  engineer  or  architect  working  on  a  DoD  program, 
putting  out  day-to-day  fires  and  still  having  to  consider  and  deal 
with  a  variety  of  technical  problems  at  depth.  Today,  they  have 
no  easy  means  to  explore  if  someone  has  already  encountered 
similar  problems  and  therefore  may  have  some  unique  insights 
to  offer.  SPRUCE  enables  sharing  of  problems  and  insights  by 
giving  the  engineer  or  architect  a  platform  to  articulate  specific 
problems,  using  concrete  artifacts  and  repeatable  experiments. 
By  posting  the  challenge  problem,  associated  artifacts  and 
experiments  in  SPRUCE,  the  engineer  can  also  help  future 
programs  that  could  face  the  same  problem. 

Next,  consider  a  software  producibility  researcher  or  graduate 
student,  focused  on  building  exciting  technologies  and  tools. 
Today,  there  is  no  readily  accessible  repository  of  ‘real-world’ 
challenge  problems  and  data  on  which  they  can  demonstrate 
and  validate  their  technologies.  Furthermore,  such  artifacts  may 
give  researchers  more  insight  into  actual  challenge  problems, 
helping  to  prevent  them  from  making  incorrect  assumptions. 
SPRUCE  provides  access  to  such  a  repository  and  also  enables  a 
perspective  on  how  their  tools  and  technologies  may  ultimately 
find  their  way  into  practice. 


spruce’s  vision  is  to  satisfy  this  multitude  of  stakeholder 
needs  by  bringing  them  together  through  (a)  well-defined,  in- 
depth  challenge  problems  and  program-representative  artifacts; 
and  (b)  repeatable  benchmarks  and  experiments  that  can  be 
readily  conducted  in  an  attached  experimentation  facility. 

Figure  1  reflects  the  current  process  within  the  DoD 
ecosystem  for  identifying,  developing,  and  transitioning  software 
producibility  technology.  Government  personnel  working  DoD 
acquisition  programs  coordinate  with  government  personnel 
working  research  programs  to  define  software  producibility 
problems  and  research  agendas.  The  problems  are  then 
described  and  written  into  research  programs’  Broad  Agency 
Announcements  (BAAs)  and  performers  are  asked  to  bid  specific 
development  and  transition  plans  for  software  producibility 
solutions. 


Figure  1:  Current  technology  identification,  development,  and 
transition  process 


Software  producibility  researchers  are  then  awarded  contracts 
to  develop  their  technology.  Unfortunately,  these  researchers 
typically  have  little  or  no  relationship  with  engineers  in  the 
program  or  domain  from  which  their  particular  challenge 
problem  is  derived.  While  researchers  strive  to  understand  and 
incorporate  deep,  specific  knowledge  about  a  problem  domain,  it 
is  hard  for  them  to  obtain  detailed  information  and  even  harder 
when  classification  and  International  Traffic  in  Arms  Regulations 
(ITAR)  issues  are  involved.  Researchers  thus  have  little  choice  but 
to  design  and  conduct  experiments  that  are  abstract  and  typically 
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small-scale  representations  of  the  real  challenge  problem.  While 
these  results  may  show  the  promise  of  the  new  technology  they 
leave  a  large  “credibility  gap”  in  the  minds  of  program  engineers 
about  whether  the  results  will  transition  into  the  real  problem 
domain.  History  indicates  that  it  is  hard  to  successfully  bridge 
this  gap,  leading  to  the  “valley  of  disappointment”  shown  in 
Figure  1 .  The  ultimate  success  or  failure  of  technology  transition 
thus  depends  on  the  ad  hoc,  opportunistic  transition  process 
described  above  where  serendipity  of  the  right  people  being  in 
the  right  positions  is  the  primary  enabler  for  success. 

The  primary  objective  of  SPRUCE  is  to  address  the  technology 
transition  problem  and  bridge  the  “valley  of  disappointment” 
described  above.  SPRUCE  emphasizes  artifacts  (e.g.,  sanitized 
DoD  application  software,  computational  resources  such  as 
specialized  avionics  processors  and  workflow  management  tools 
and  services),  typically  provided  in  the  context  of  challenge 
problems,  and  experimentation  around  them  to  create  a 
common  clearinghouse  for  program  engineers  and  technology 
researchers  to  discover  joint  interests  and  form  collaborations. 
Collaborations  on  real  world  software  producibility  challenges 
and  the  associated  experiments  using  realistic  artifacts  are  the 
key  to  successful  technology  transition. 

What  is  SPRUCE? 

SPRUCE  is  an  open,  collaborative  research  and  development 
environment  to  demonstrate,  evaluate,  and  document  the 
ability  of  novel  tools,  methods,  techniques,  and  run-time 
technologies  to  yield  affordable  and  more  predictable  production 
of  software-intensive  systems.  The  key  elements  of  SPRUCE 
are:  (1)  a  collaboration  environment  that  enables  and  sustains 
active  collaboration  between  various  stakeholders,  allowing 
stakeholders  to  describe  and  discuss  challenge  problems, 
potential  solutions,  and  experiments  to  showcase  the  problems 
and  evaluate  solutions,  and  (2)  an  experimentation  environment 
containing  realistic  software  artifacts  and  computing  systems 
that  promote  scientifically  rigorous  evaluation.  The  following 
sections  describe  the  key  SPRUCE  elements  in  more  detail. 

SPRUCE  Collaboration  Environment 

The  SPRUCE  collaboration  environment,  implemented 
as  a  web  portal,  seeks  to  empower  its  users  to  define  and 
evolve  narrow,  well-defined  technology  problems  of  mutual 
interest — but  at  depth — and  seeks  to  provide  them  with  tools 
for  collaboration  and  discovery.  To  achieve  this  goal,  SPRUCE 
structures  its  collaboration  environment  around  four  basic 
concepts:  communities  of  interest  (Col),  challenge  problems, 
candidate  solutions,  and  experiments  and  experiment  instances 
(Figure  2). 

2 


Figure  2:  Key  SPRUCE  Concepts 


Communities  of  Interest  (Col):  Communities  of  interest 
serve  to  organize  SPRUCE  content  (i.e.,  challenge  problems, 
candidate  solutions  and  associated  discussions)  around  broad 
but  focused  topic  areas.  They  also  serve  as  a  virtual  meeting  place 
for  SPRUCE  users.  SPRUCE  users  can  belong  to  one  or  more 
communities  of  interest. 

Challenge  Problems:  SPRUCE  challenge  problems  represent 
sanitized  versions  of  realistic  problems  that  may  occur  on  actual 
DoD  acquisition  programs.  These  problems  may  have  occurred 
on  other  DoD  programs  in  the  past,  may  express  a  desire  to 
solve  future  anticipated  problems  that  would  be  tedious  to  solve 
using  existing  means,  or  may  provide  a  context  for  radically 
new  approaches  to  systems  and  software  development.  As  these 
challenge  problems  represent  a  shared  concern,  they  provide 
an  opportunity  to  bring  together  the  various  stakeholders 
in  the  DoD  software-intensive  systems  producibility  (SISP) 
ecosystem.  SPRUCE  encourages  and  enables  DoD  programs  to 
submit  realistic  and  sanitized  artifacts  that  accompany  challenge 
problems  to  attract  researchers  and  provide  real-world  depth  for 
challenge  problems. 

Candidate  Solutions:  SPRUCE  candidate  solutions  describe 
proposed  solutions  to  SPRUCE  challenge  problems.  Since 
SPRUCE  challenge  problems  represent  realistic  problems  faced 
by  DoD  programs,  successful  SPRUCE  candidate  solutions  are 
more  amenable  to  technology  transfer.  Researchers  and  tool 
vendors  may,  if  desired,  elect  to  upload  their  technology  and 
tools  into  SPRUCE  and  to  associate  licensing  conditions  with 
the  use  of  the  tools.  More  likely,  however,  SPRUCE  will  be  used 
to  highlight  specific  properties  of  the  tools  and  solutions  and  how 
they  address  specific  challenge  problems  posed.  Researchers  and 
tool  vendors  can  provide  links  to  their  solutions  for  interested 
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SPRUCE  users  to  access. 

Experiments:  SPRUCE  experiments  are  associated  with 
challenge  problems  and  candidate  solutions,  and  serve  two 
primary  purposes:  (a)  to  showcase  scenarios  described  in  a 
challenge  problem,  so  that  SPRUCE  community  members 
have  a  repeatable  baseline  or  (b)  to  evaluate  the  effectiveness  of 
a  particular  solution  or  set  of  solutions  against  a  benchmark. 
In  the  former  case,  they  are  best  initiated  and  mediated  by  the 
challenge  problem  provider,  whereas  a  solution  provider  is  best 
suited  to  define  and  conduct  the  latter  kinds  of  experiments. 
Experiment  instances  represent  an  instantiation  of  a  SPRUCE 
experiment  that  can  be  run  on  actual  hardware,  including  the 
SPRUCE  experimentation  environment  (discussed  in  the  next 
section). 

As  shown  in  Figure  2,  challenge  problems,  candidate  solutions 
and  experiments  are  interrelated,  and  each  can  belong  to  one 
or  more  communities  of  interest.  To  facilitate  a  community’s 
access  to  collaboration,  SPRUCE  automatically  creates  artifact 
repositories,  community  wiki  and  discussion  fora  (termed 
collaboration  facilities’)  for  each  of  these  entities  and  makes 
them  readily  accessible  from  the  entity’s  main  page.  The  use  of 
social  networking  tools  and  instant  communication  facilities, 
such  as  rich  text  and  media  chat,  as  well  as  member  presence 
information  is  being  considered  for  future  capabilities. 

SPRUCE  Experimentation  Environment 

In  addition  to  the  web  portal,  SPRUCE  provides  an 
experimentation  environment  that  is  available  to  all  SPRUCE 
users.  This  environment,  comprised  of  real  hardware  resources, 
can  be  used  to  illustrate  challenge  problems  and  showcase 
candidate  solutions  in  a  repeatable  manner  on  a  representative 
environment.  The  SPRUCE  experimentation  environment  is 
based  on  Emulab  (www.emulab.net). 

SPRUCE  users  can  access  the  experimentation  environment 
remotely,  request  and  receive  experiment  resources,  setup  desired 
experiment  configurations,  download  specific  operating  systems 
and  software,  conduct  experiments,  and  collect  results.  This 
interaction  is  done  manually  with  the  help  of  scripts.  Results 
of  the  experiments  can  be  posted  to  SPRUCE  wiki  pages  to  be 
shared  with  other  SPRUCE  users.  Future  work  will  enable  the 
seamless  integration  of  the  experimentation  environment  and 
the  collaboration  environment,  allowing  reuse,  cataloging,  and 
automation  of  many  of  these  functions.  The  use  of  a  community 
wiki  to  post  and  discuss  experimental  results  enables  community- 
driven  peer  review  and  discussion,  much  like  Wikipedia®, 
thereby  enhancing  the  credibility  of  the  content. 


Figure  3  shows  the  current  hardware  architecture  of  the 
SPRUCE  experimentation  environment  with  standard  blades 
and  network  switches  that  serve  the  reconfigurable  nature  of 
the  experimentation  environment.  This  figure  also  shows  where 
specialized  equipment  can  be  connected  and  provisioned. 
SPRUCE  will  formulate  an  equipment  donation  program 
through  which  legacy  and  specialized  equipment  can  be  made 
available  for  use. 
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Figure  3:  SPRUCE  experimentation  environment  is  based  on  the  Emulab 
technology 


Examples  in  SPRUCE 

The  SPRUCE  portal  (http://www.sprucecommunity.org) 
is  currently  in  Beta  phase  with  a  limited  number  of  registered 
users  validating  use  cases  and  collaborating  around  an  initial 
set  of  challenge  problems  associated  with  an  initial  set  of 
communities  of  interest.  Current  SPRUCE  communities  of 
interest  include:  (1)  real-time  and  embedded  systems,  (2)  multi¬ 
core  architectures,  and  (3)  feature-oriented  software  analytics. 
The  screen  images  shown  in  Figures  4,  5,  6  and  7  showcase 
initial  challenge  problems,  experiments,  and  collaborations 
posted  in  SPRUCE. 

Figure  4  illustrates  the  main  page  of  a  challenge  problem 
that  serves  as  the  landing  page  for  the  community  interested  in 
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Figure  4:  SPRUCE  challenge  problem  main  page  for  the  cache  false  sharing  challenge  problem 


this  problem.  The  main  pages  of  other  SPRUCE  components 
are  structured  similarly.  Each  challenge  problem  has  associated 
metadata:  title,  description,  sponsors,  keywords  and  a  collection 
of  Cols  (label  1  in  Figure  4).  There  are  dedicated  collaboration 
facilities,  such  as  discussion  forum  topics,  wiki  pages,  and  artifact 
repositories  associated  with  a  challenge  problem  (label  2  in 
Figure  4).  Lists  and  hotlinks  to  related  SPRUCE  entities  such 
as  experiments,  challenge  problems,  and  candidate  solutions 
are  also  available  for  easy  navigation  and  cross-reference  (label 
3  in  Figure  4). 


Figure  4  shows  a  sample  challenge  problem  related  to  “cache 
false-sharing”  in  multi-core  architectures.  In  this  problem, 
conflicting  cache  requirements  of  programs  running  on  multiple 
processor  cores  constantly  invalidate  the  processor  cache  and 
thus  cause  performance  degradation  by  defeating  the  purpose 
for  which  the  cache  was  designed.  This  challenge  problem  is 
part  of  the  “multi-core  architectures”  Col. 
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Cache  False-Sharing  Benchmark  E^iperiment 


Summary  oF  Results 
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Figure  5:  SPRUCE  experiment  wiki  for  the  “cache  false-sharing”  problem 


Figure  5  shows  a  wiki  entry  from  an  experiment  conducted  for  the  “cache  false-sharing”  problem  of  Figure  4.  The  wiki  allows 
for  free-form  analysis  of  experiment  results  and  generation  of  potential  ideas  for  future  research.  SPRUCE  users  editing  the  wiki 
page  can  also  create  links  to  other  places  in  the  SPRUCE  portal  for  easy  navigation. 
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Figure  6:  Example  artifact  for  the  multi-dimensional  resource  optimization  problem 


Figure  6  shows  an  artifact  associated  with  another  challenge  problem  in  multi-dimensional  resource  optimization.  The  data  set 
is  a  sanitized  version  of  an  artifact  from  a  military  and  aeronautics  application  and  represents  1 1 ,406  types  of  network  messages 
flowing  across  46  processors.  Each  messages  size,  frequency,  semantic  type,  etc.,  are  specifled  as  shown  in  the  top  pane.  The  CPU 
loading  for  14  of  the  processors,  which  are  available  for  allocation,  are  also  specifled.  Collaborations  are  currently  in  progress  to 
develop  further  assumptions,  experiments,  and  solutions  to  this  challenge  problem.  Since  the  challenge  problem  provider  also 
serves  as  an  active  moderator  for  the  collaborations,  the  assumptions,  and  experiments  are  guided  with  an  eye  toward  technology 
application  in  the  target  environment. 


an 


Figure  7  shows  a  screen  snapshot  from  the  wiki  page  of  an  experiment  conducted  with  a  candidate  solution  (using  ASCENT, 
algorithm  from  Vanderbilt  University)  that  was  used  to  analyze  a  subset  of  the  problem  in  Figure  6. 
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Results' 


Downloading  Results: 

The  full  results  evaluation  is  available  in  Excel  form  frOiri;  https  :v7'.vVi'w.5O'uce:;cT;mun>.v.crin/rai:ieb.-Pii0'''^2CJe3'jjvmer  L'^AZUCDb  rr!^zdl:i3~/e.^r:j  sci-i.iun  evalLidLion.xIsx. 


Results  Summary: 
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Figure  7:  Example  screen  from  a  candidate  solution  documenting  the  results  of  an  experiment 


Enhancing  Community  Building 

Foundational  technology  to  support  the  discovery  of  suitable 
participants  in  the  various  SPRUCE  communities  is  currently 
under  development.  This  technology  is  based  on  the  notion  of 
affinity  relationships  between  challenge  problems,  solutions 
technologies,  and  personnel  interests  and  publications.  The 
automated  collection  and  construction  of  affinity  relationships 
will  allow  SPRUCE  to  offer  query  mechanisms  such  as  “since 
person  A  is  interested  in  this  problem,  who  else  may  be 
interested?”,  or  “given  the  challenge  problems  description, 
construct  a  list  of  leading  researchers  that  have  published  in 
specified  leading  journals  on  related  subjects.”  SPRUCE  users 
can  then  use  the  results  of  such  queries  to  construct  invitations 
to  potential  collaborators. 


Other  features  under  consideration  include  social  networking 
and  community  communication  mechanisms  to  facilitate 
a  virtual  community,  such  as  instant  messaging,  presence 
information,  text  and  media  chat. 

Getting  Involved 

If  you  wish  to  participate  in  SPRUCE,  please  visit  the 
SPRUCE  portal  (www.sprucecommunity.org)  and/or  contact 
the  authors.  All  forms  of  participation  and  contributions  are 
welcome  -  be  it  through  participating  with  existing  communities 
of  interest,  establishing  and  leading  new  communities  of  interest, 
or  providing  representative  DoD  artifacts  to  be  shared  with  the 
SPRUCE  community. 
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Conclusion 

Experimentation  and  collaboration  around  representative 
challenge  problems  in  military  and  aeronautics  domains  have 
the  potential  to  bridge  the  divide  between  various  stakeholders 
in  the  DoD  ecosystem — a  challenge  heretofore  achieved 
only  through  ad  hoc  and  serendipitous  engagement  between 
participants.  SPRUCE  offers  an  exciting  opportunity  to  address 
this  challenge  head  on,  by  providing  a  platform  to  promote 
the  desired  experimentation  and  collaboration.  The  SPRUCE 
program  will  undertake  a  foundational  effort  in  getting  an 
initial  set  of  communities  started.  Members  of  the  SPRUCE 
community  have  the  power  to  sustain  and  shape  the  evolution 
of  SPRUCE  in  the  years  ahead. 
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